The present invention relates to an image sensing device, an image processing apparatus and method, and a memory medium.
FIG. 16 is a schematic block diagram showing the arrangement of a conventional digital still camera. In this digital camera, an overall control circuit 80 detects changes in state of operation switches (main switch, release switch, and the like) 91, and starts power supply to other blocks.
An object image within the photographing frame range is formed on the image sensing surface of an image sensing unit 82 via a main photographing optical system 81. The image sensing unit 82 converts this image into an analog electrical signal, and supplies it to an A/D converter 83 in turn in units of pixels. The A/D converter 83 converts the analog electrical signal into a digital signal, and supplies the digital signal to a process circuit 84.
The process circuit 84 generates R, G, and B image signals on the basis of the input data. In the state before photographing, these image signals are periodically transferred to a video memory 89 via a memory controller 85 in units of frames, thus displaying an image on a display unit 90. In this manner, a viewfinder display is implemented.
On the other hand, when the photographer operates one of operation switches 91 to instruct execution of photographing, pixel data for one frame output from the process circuit 84 are stored in a frame memory 86 in accordance with a control signal output from the overall control circuit 80. The data in the frame memory 86 are compressed by the memory controller 85 and a work memory 87 on the basis of a predetermined compression format, and the compression result is stored in an external memory (e.g., a nonvolatile memory such as a flash memory or the like) 88.
When the photographer reviews photographed images, the compressed data stored in the external memory 88 are read out, and are expanded by the memory controller 85 to be converted into normal data in units of photographing pixels. The conversion result is transferred to the video memory 89, and an image is displayed on the display unit 90.
In such digital camera, microlenses 31 shown in FIGS. 2A and 2B are provided to an image sensing element of the image sensing unit 32 in units of photosensitive pixels so as to improve optical sensitivity in units of pixels. In FIGS. 2A and 2B, reference numeral 30 denotes a photographing lens (main photographing optical system) as a field lens; and 32, each photosensitive section (light-receiving section) of the image sensing element.
Since the microlenses 31 are provided in units of photosensitive pixels of the image sensing element, even when the image sensing element has a narrow effective sensitivity range, marginal light can be effectively focused on photosensitive pixels.
When light rays have passed through the photographing lens 30 strike the image sensing element in a direction nearly parallel to the optical axis, as shown in FIG. 2A, incoming light rays focus on the photosensitive section 31 without posing any serious problem. However, when light rays obliquely enter the photographing lens 30, as shown in FIG. 2B, only some of the incoming light rays become incident on the photosensitive section 31 in an area (peripheral portion of the image sensing element) separated from the optical axis of the photographing lens 30.
Such light amount drop is normally called white shading. This phenomenon becomes conspicuous as the pixel position on the image sensing element separates farther away from the optical axis of the photographing lens, and also as the focal length of the photographing lens 30 decreases (in practice, as the distance from the image sensing element to the pupil position of the photographing lens becomes shorter).
FIGS. 3A and 3B are graphs showing changes in white shading when the stop of the photographing lens 30 has changed. FIGS. 3A and 3B respectively show the relationship between the image height and relative sensitivity on the image sensing element in a full-aperture state (FIG. 3A), and a stop-down state (FIG. 3B). In the full-aperture state of the stop, the relative sensitivity drops largely compared to the central portion with increasing height of an image formed on the image sensing element, since many light components that obliquely enter the photographing optical system are included, as shown in FIG. 2B.
On the other hand, when the stop is stopped down, the relative sensitivity changes little even when the image height becomes larger, since light components that obliquely enter the photographing lens 30 are suppressed by the stop effect.
As a method of correcting white shading produced by a combination of the photographing lens 30 and microlenses on the image sensing element, a method disclosed in Japanese Patent Laid-Open No. 9-130603 is known. In this method, assuming that each pixel position on the image sensing element is expressed by a pointer indicated by a horizontal direction X and vertical direction Y, shading correction data H(x) for one horizontal line, and shading correction data V(y) for one vertical line are used. The shading correction data for one horizontal line and one vertical line assume larger values as the pixel position approaches an end (periphery). A pixel S(i, j) on the image sensing element is multiplied by horizontal correction data H(i) and vertical correction data V(j) as per:
S(i, j)xc3x97H(j)xc3x97V(j)xe2x86x92S(i, j)
In this way, sensitivity drop in the peripheral portion of the image sensing element due to shading is apparently prevented.
When a digital camera is constructed using the same optical system (e.g., exchangeable lens system) as that of a conventional single-lens reflex silver halide camera, an image sensing element having an image sensing area considerably larger than that of a normal image sensing element is required.
However, an image sensing element having such a large image sensing area often suffers a problem of sensitivity nonuniformity of color filters and the like, which is produced in the manufacture process. In view of the whole image sensing element, only a local area suffers a decrease or increase in sensitivity, which cannot be corrected by simply setting higher gains toward the periphery.
In FIG. 4, 401 indicates an example of the sensitivity distribution of the image sensing element, in which the central sensitivity is higher than the peripheral one. Such sensitivity nonuniformity can be corrected to some extent using the aforementioned conventional method.
For example, 402 in FIG. 4 expresses a function of horizontal sensitivity correction data H(i), which has smaller correction coefficient values at the central portion and larger ones at the periphery. On the other hand, 403 in FIG. 4 expresses a function of vertical sensitivity correction data V(j), which has smaller correction coefficient values at the central portion and larger ones at the periphery.
Therefore, correction can be done using such functions by calculating a value S(i, j) at each pixel point as per:
S(i, j)xc3x97H(i)xc3x97V(j)xe2x86x92S(i, j)
However, when local sensitivity nonuniformity is present in a plurality of areas on the frame, as indicated by 501 in FIG. 5, it cannot be perfectly removed by the above-mentioned correction method that simply uses shading correction data H(x) for one horizontal line and shading correction data V(y) for one vertical line.
On the other hand, in the device structure of an image sensing element indicated by 601 in FIG. 6, since aluminum interconnects and the like run to sandwich, e.g., two sensitivity areas, the level of incoming light to one (G sensitivity area) of two sensitivity areas lowers in areas on the left side in FIG. 6, while one (R sensitivity area) of two sensitivity areas lowers in areas on the right side in FIG. 6. Therefore, the two sensitivity areas (G and R) have different sensitivity levels depending on the right and left areas of the frame of the image sensing element, as indicated by 602 in FIG. 6.
Likewise, in neighboring rows (rows in which G and R alternately appear), the sensitivity distribution changes, as indicated by 603 in FIG. 6. That is, in this case, since a Bayer matrix is assumed as a matrix of color filters, G sensitivity lowers to the right unlike the characteristics indicated by 602.
When colors are formed by combining R, G, and B with such characteristics, different color tones are obtained on the right, left, up, and down positions on the frame. Such phenomenon is generally called color shading, which is also produced by the aforementioned combination of the photographing lens and microlenses, in addition to the device structure.
Against such phenomenon, color unbalances cannot be sufficiently corrected by only combining the conventional linear correction data sequences H(i) and V(j).
The present invention has been made in consideration of the above situation, and has as its object to correct, e.g., sensitivity nonuniformity of an image sensing element.
More specifically, it is an object of the present invention to correct, e.g., local sensitivity nonuniformity of an image sensing element resulting from variations or the like of color filters provided to the image sensing element in the manufacture process.
It is another object of the present invention to correct sensitivity nonuniformity (e.g., color shading, white shading) of an image sensing element resulting from a combination of the device structure of the image sensing element and a photographing optical system.
It is still another object of the present invention to correct sensitivity nonuniformity (e.g., color shading, white shading) of an image sensing element resulting from a combination of the device structure of the image sensing element and a photographing optical system independently from a change in state of the photographing optical system.
Other objects of the present invention will be described in the embodiments of the present invention.
An image sensing device according to the first aspect of the present invention comprises an image sensing element having a two-dimensional matrix of pixels, first multiplier means for multiplying a row of an image sensed by the image sensing element by horizontal linear correction data, second multiplier means for multiplying a column of the image by vertical linear correction data, and change means for changing at least one of the horizontal and vertical linear correction data in accordance with a position of a pixel of interest in the image, and the first and second multiplier means correct a value of each pixel of the image.
In the image sensing device according to the first aspect of the present invention, for example, the image is preferably segmented into a plurality of partial areas, and the change means changes at least one of the horizontal and vertical linear correction data in accordance with the partial area to which the pixel of interest belongs.
In the image sensing device according to the first aspect of the present invention, for example, the image is preferably segmented into a plurality of partial areas, and the change means changes both the horizontal and vertical linear correction data in accordance with the partial area to which the pixel of interest belongs.
In the image sensing device according to the first aspect of the present invention, for example, the change means preferably has two different types of horizontal linear correction data, alternately switches the horizontal linear correction data in units of rows, and supplies the selected data to multiplication by the first multiplier means.
In the image sensing device according to the first aspect of the present invention, for example, the change means preferably has two different types of vertical linear correction data, alternately switches the vertical linear correction data in units of columns, and supplies the selected data to multiplication by the second multiplier means.
In the image sensing device according to the first aspect of the present invention, for example, the image sensing element preferably includes color filters in a Bayer matrix.
In the image sensing device according to the first aspect of the present invention, for example, the image is preferably segmented into a plurality of groups each of which is defined by an area as a two-dimensional matrix of a plurality of pixels, the device further comprises third multiplier means for multiplying each group by two-dimensional correction data given by a row and column, and the change means includes means for changing the two-dimensional correction data in accordance with the group to which the pixel of interest in the image belongs.
In the image sensing device according to the first aspect of the present invention, for example, the change means preferably includes means for changing at least one of the horizontal and vertical linear correction data in accordance with a state of a photographing optical system.
In the image sensing device according to the first aspect of the present invention, for example, the change means preferably includes means for changing at least one of the horizontal and vertical linear correction data in accordance with one of a focal length, field angle, and aperture of a photographing optical system or a combination thereof.
An image processing apparatus according to the second aspect of the present invention is an image processing apparatus for processing an image supplied from an image sensing element having a two-dimensional matrix of pixels, comprising first multiplier means for multiplying a row of an image sensed by the image sensing element by horizontal linear correction data, second multiplier means for multiplying a column of the image by vertical linear correction data, and change means for changing at least one of the horizontal and vertical linear correction data in accordance with a position of a pixel of interest in the image, wherein the first and second multiplier means correct a value of each pixel of the image.
An image processing method according to the third aspect of the present invention comprises the first multiplier step of multiplying a row of an image sensed by an image sensing element having a two-dimensional matrix of pixels by horizontal linear correction data, the second multiplier step of multiplying a column of the image by vertical linear correction data, and the change step of changing at least one of the horizontal and vertical linear correction data in accordance with a position of a pixel of interest in the image, and a value of each pixel of the image is corrected in the first and second multiplier steps.
A memory medium according to the fourth aspect of the present invention is a memory medium storing an image processing program, the program including the first multiplier step of multiplying a row of an image sensed by an image sensing element having a two-dimensional matrix of pixels by horizontal linear correction data, the second multiplier step of multiplying a column of the image by vertical linear correction data, and the change step of changing at least one of the horizontal and vertical linear correction data in accordance with a position of a pixel of interest in the image, wherein a value of each pixel of the image is corrected in the first and second multiplier steps.